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The Sampling Theorem 
Converting between a signal and numbers. 


Analog-to-Digital Conversion 


Because of the way computers are organized, signal must be represented by 
a finite number of bytes. This restriction means that both the time axis and 
the amplitude axis must be quantized: They must each be a multiple of the 
integers. [footnote] Quite surprisingly, the Sampling Theorem allows us to 
quantize the time axis without error for some signals. The signals that can 
be sampled without introducing error are interesting, and as described in the 
next section, we can make a signal "samplable" by filtering. In contrast, no 
one has found a way of performing the amplitude quantization step without 
introducing an unrecoverable error. Thus, a signal's value can no longer be 
any real number. Signals processed by digital computers must be discrete- 
valued: their values must be proportional to the integers. Consequently, 
analog-to-digital conversion introduces error. 

We assume that we do not use floating-point A/D converters. 


The Sampling Theorem 


Digital transmission of information and digital signal processing all require 
signals to first be "acquired" by a computer. One of the most amazing and 
useful results in electrical engineering is that signals can be converted from 
a function of time into a sequence of numbers without error: We can 
convert the numbers back into the signal with (theoretically) no error. 
Harold Nyquist, a Bell Laboratories engineer, first derived this result, 
known as the Sampling Theorem, in the 1920s. It found no real application 
back then. Claude Shannon, also at Bell Laboratories, revived the result 
once computers were made public after World War II. 


The sampled version of the analog signal s(t) is s(nT,), with T, known as 
the sampling interval. Clearly, the value of the original signal at the 

sampling times is preserved; the issue is how the signal values between the 
samples can be reconstructed since they are lost in the sampling process. To 
characterize sampling, we approximate it as the product x(t) = s(t)Pr.(t), 


with Pr, (t) being the periodic pulse signal. The resulting signal, as shown 
in [link], has nonzero values only during the time intervals 

(nT, — 4,nT,+ 4),ne€ {...,-1,0,1,...}. 

Sampled Signal 
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The waveform of an example signal 
is shown in the top plot and its 
sampled version in the bottom. 


For our purposes here, we center the periodic pulse signal about the origin 
so that its Fourier series coefficients are real (the signal is even). 
Equation: 


Equation: 


If the properties of s(t) and the periodic pulse signal are chosen properly, 
we can recover s(t) from x(t) by filtering. 


To understand how signal values between the samples can be "filled" in, we 
need to calculate the sampled signal's spectrum. Using the Fourier series 
representation of the periodic sampling signal, 

Equation: 
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Considering each term in the sum separately, we need to know the spectrum 
of the product of the complex exponential and the signal. Evaluating this 
transform directly is quite easy. 


Equation: 
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Thus, the spectrum of the sampled signal consists of weighted (by the 
coefficients c;) and delayed versions of the signal's spectrum ([link]). 
Equation: 
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In general, the terms in this sum overlap each other in the frequency 
domain, rendering recovery of the original signal impossible. This 
unpleasant phenomenon is known as aliasing. 

aliasing 


The spectrum of some bandlimited (to W Hz) 
signal is shown in the top plot. If the sampling 
interval 7’, is chosen too large relative to the 
bandwidth W, aliasing will occur. In the 
bottom plot, the sampling interval is chosen 
sufficiently small to avoid aliasing. Note that 
if the signal were not bandlimited, the 
component spectra would always overlap. 


If, however, we satisfy two conditions: 


e The signal s(t) is bandlimited—has power in a restricted frequency 
range—to W Hz, and 

e the sampling interval 7’, is small enough so that the individual 
components in the sum do not overlap— T, < 1/2W, 


aliasing will not occur. In this delightful case, we can recover the original 
signal by lowpass filtering x(t) with a filter having a cutoff frequency equal 
to W Hz. These two conditions ensure the ability to recover a bandlimited 
signal from its sampled version: We thus have the Sampling Theorem. 
Exercise: 


Problem: 


The Sampling Theorem (as stated) does not mention the pulse width A 
. What is the effect of this parameter on our ability to recover a signal 
from its samples (assuming the Sampling Theorem's two conditions 
are met)? 


Solution: 


The only effect of pulse duration is to unequally weight the spectral 
repetitions. Because we are only concemed with the repetition 
centered about the origin, the pulse duration has no significant effect 
on recovering a signal from its samples. 


The frequency =4-, known today as the Nyquist frequency and the 
Y or 


Shannon sampling frequency, corresponds to the highest frequency at 
which a signal can contain energy and remain compatible with the 
Sampling Theorem. High-quality sampling systems ensure that no aliasing 
occurs by unceremoniously lowpass filtering the signal (cutoff frequency 
being slightly lower than the Nyquist frequency) before sampling. Such 
systems therefore vary the anti-aliasing filter's cutoff frequency as the 
sampling rate varies. Because such quality features cost money, many 
sound cards do not have anti-aliasing filters or, for that matter, post- 
sampling filters. They sample at high frequencies, 44.1 kHz for example, 
and hope the signal contains no frequencies above the Nyquist frequency 
(22.05 kHz in our example). If, however, the signal contains frequencies 
beyond the sound card's Nyquist frequency, the resulting aliasing can be 
impossible to remove. 

Exercise: 


Problem: 


To gain a better appreciation of aliasing, sketch the spectrum of a 
sampled square wave. For simplicity consider only the spectral 
repetitions centered at — a 0, a Let the sampling interval 7’, be 1; 
consider two values for the square wave's period: 3.5 and 4. Note in 
particular where the spectral lines go as the period decreases; some 
will move to the left and some to the right. What property 
characterizes the ones going the same direction? 


Solution: 


The square wave's spectrum is shown by the bolder set of lines 
centered about the origin. The dashed lines correspond to the 
frequencies about which the spectral repetitions (due to sampling with 
T’, = 1) occur. As the square wave's period decreases, the negative 
frequency lines move to the left and the positive frequency ones to the 
right. 


If we satisfy the Sampling Theorem's conditions, the signal will change 
only slightly during each pulse. As we narrow the pulse, making A smaller 
and smaller, the nonzero values of the signal s(t)pr,(t) will simply be 
s(nT, ), the signal's samples. If indeed the Nyquist frequency equals the 
signal's highest frequency, at least two samples will occur within the period 
of the signal's highest frequency sinusoid. In these ways, the sampling 
signal captures the sampled signal's temporal variations in a way that leaves 
all the original signal's structure intact. 


Exercise: 


Problem: 


What is the simplest bandlimited signal? Using this signal, convince 
yourself that less than two samples/period will not suffice to specify it. 
If the sampling rate a is not high enough, what signal would your 


resulting undersampled signal become? 
Solution: 


The simplest bandlimited signal is the sine wave. At the Nyquist 
frequency, exactly two samples/period would occur. Reducing the 
sampling rate would result in fewer samples/period, and these samples 
would appear to have arisen from a lower frequency sinusoid. 


Amplitude Quantization 
Analog-to-digital conversion. 


The Sampling Theorem says that if we sample a bandlimited signal s(t) 
fast enough, it can be recovered without error from its samples s(nT;), 

n € {...,—1,0,1,...}. Sampling is only the first phase of acquiring data 
into a computer: Computational processing further requires that the samples 
be quantized: analog values are converted into digital form. In short, we 
will have performed analog-to-digital (A/D) conversion. 


A three-bit A/D converter assigns voltage in the range [—1, 1] to one 
of eight integers between 0 and 7. For example, all inputs having 
values lying between 0.5 and 0.75 are assigned the integer value six 
and, upon conversion back to an analog value, they all become 0.625. 
The width of a single quantization interval A equals *. The bottom 


panel shows a signal going through the analog-to-digital converter, 
where B is the number of bits used in the A/D conversion process (3 in 
the case depicted here). First it is sampled, then amplitude-quantized to 
three bits. Note how the sampled signal waveform becomes distorted 
after amplitude quantization. For example the two signal values 
between 0.5 and 0.75 become 0.625. This distortion is irreversible; it 
can be reduced (but not eliminated) by using more bits in the A/D 
converter. 


signal sampled signal amplitude-quantzed 
and sampled signal 
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A phenomenon reminiscent of the errors incurred in representing numbers 
on a computer prevents signal amplitudes from being converted with no 
error into a binary number representation. In analog-to-digital conversion, 
the signal is assumed to lie within a predefined range. Assuming we can 
scale the signal without affecting the information it expresses, we'll define 
this range to be [—1, 1]. Furthermore, the A/D converter assigns amplitude 
values in this range to a set of integers. A B-bit converter produces one of 
the integers {0, 1, i052? = 1} for each sampled input. [link] shows how a 
three-bit A/D converter assigns input values to the integers. We define a 
quantization interval to be the range of values assigned to the same 
integer. Thus, for our example three-bit A/D converter, the quantization 
interval A is 0.25; in general, it is ss. 

Exercise: 


Problem: 
Recalling the plot of average daily highs in this frequency domain 


problem, why is this plot so jagged? Interpret this effect in terms of 
analog-to-digital conversion. 


Solution: 


The plotted temperatures were quantized to the nearest degree. Thus, 
the high temperature's amplitude was quantized as a form of A/D 
conversion. 


Because values lying anywhere within a quantization interval are assigned 
the same value for computer processing, the original amplitude value 
cannot be recovered without error. Typically, the D/A converter, the 
device that converts integers to amplitudes, assigns an amplitude equal to 
the value lying halfway in the quantization interval. The integer 6 would be 
assigned to the amplitude 0.625 in this scheme. The error introduced by 
converting a signal from analog to digital form by sampling and amplitude 
quantization then back again would be half the quantization interval for 
each amplitude value. Thus, the so-called A/D error equals half the width of 
a quantization interval: sr. As we have fixed the input-amplitude range, 


the more bits available in the A/D converter, the smaller the quantization 
error. 


To analyze the amplitude quantization error more deeply, we need to 
compute the signal-to-noise ratio, which equals the ratio of the signal 
power and the quantization error power. Assuming the signal is a sinusoid, 
the signal power is the square of the rms amplitude: 


2 
power (s) = (4) = +. The illustration details a single quantization 
interval. 
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Its width is A and the quantization error is denoted by e. To find the power 
in the quantization error, we note that no matter into which quantization 
interval the signal's value falls, the error will have the same characteristics. 
To calculate the rms value, we must square the error and average it over the 
interval. 

Equation: 


rms (€) 5 fiserde 


Since the quantization interval width for a B-bit converter equals 


Se — 2~(2-)) we find that the signal-to-noise ratio for the analog-to- 
digital conversion process equals 
Equation: 
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= 2 _ %52B _ 
SNR = ae = 5 2°" = 6B + 10log1.5dB 
12 


Thus, every bit increase in the A/D converter yields a 6 dB increase in the 
signal-to-noise ratio. The constant term 10 log 1.5 equals 1.76. 
Exercise: 


Problem: 


This derivation assumed the signal's amplitude lay in the range {[—1, 1]. 
What would the amplitude quantization signal-to-noise ratio be if it lay 
in the range [— A, A]? 


Solution: 


The signal-to-noise ratio does not depend on the signal amplitude. 
With an A/D range of [—A, A], the quantization interval A = 24 and 


the signal's rms value (again assuming it is a sinusoid) is a 
Exercise: 
Problem: 


How many bits would be required in the A/D converter to ensure that 
the maximum amplitude quantization error was less than 60 db smaller 
than the signal's peak value? 


Solution: 


Solving 2-7 = .001 results in B = 10 bits. 
Exercise: 


Problem: 


Music on a CD is stored to 16-bit accuracy. To what signal-to-noise 
ratio does this correspond? 


Solution: 


A 16-bit A/D converter yields a SNR of 6 x 164+ 10log1.5 = 97.8 
dB. 


Once we have acquired signals with an A/D converter, we can process them 
using digital hardware or software. It can be shown that if the computer 
processing is linear, the result of sampling, computer processing, and 


unsampling is equivalent to some analog linear system. Why go to all the 
bother if the same function can be accomplished using analog techniques? 
Knowing when digital processing excels and when it does not is an 
important issue. 


Discrete-Time Filtering of Analog Signals 
A brief introduction on how to filter digital signals 


Because of the Sampling Theorem, we can process, in particular filter, 
analog signals "with a computer" by constructing the system shown in 
[link]. To use this system, we are assuming that the input signal has a 
lowpass spectrum and can be bandlimited without affecting important 
signal aspects. Bandpass signals can also be filtered digitally, but require a 
more complicated system. Highpass signals cannot be filtered digitally. 
Note that the input and output filters must be analog filters; trying to 
operate without them can lead to potentially very inaccurate digitization. 


To process an analog signal digitally, the signal 
x(t) must be filtered with an anti-aliasing filter 
(to ensure a bandlimited signal) before A/D 
conversion. This lowpass filter (LPF) has a 
cutoff frequency of W Hz, which determines 
allowable sampling intervals T’,. The greater 
the number of bits in the amplitude 
quantization portion Q|-] of the A/D converter, 
the greater the accuracy of the entire system. 
The resulting digital signal z(n) can now be 
filtered in the time-domain with a difference 
equation or in the frequency domain with 
Fourier transforms. The resulting output y(7) 
then drives a D/A converter and a second anti- 
aliasing filter (having the same bandwidth as 
the first one). 


Another implicit assumption is that the digital filter can operate in real 
time: The computer and the filtering algorithm must be sufficiently fast so 


that outputs are computed faster than input values arrive. The sampling 
interval, which is determined by the analog signal's bandwidth, thus 
determines how long our program has to compute each output y(n). The 
computational complexity for calculating each output with a difference 
equation is O(p + q). Frequency domain implementation of the filter is also 
possible. The idea begins by computing the Fourier transform of a length-N 
portion of the input x(n), multiplying it by the filter's transfer function, and 
computing the inverse transform of the result. This approach seems overly 
complex and potentially inefficient. Detailing the complexity, however, we 
have O(N log N) for the two transforms (computed using the FFT 
algorithm) and O(V) for the multiplication by the transfer function, which 
makes the total complexity O(N log NV) for N input values. A frequency 
domain implementation thus requires O(log NV’) computational complexity 
for each output value. The complexities of time-domain and frequency- 
domain implementations depend on different aspects of the filtering: The 
time-domain implementation depends on the combined orders of the filter 
while the frequency-domain implementation depends on the logarithm of 
the Fourier transform's length. 


It could well be that in some problems the time-domain version is more 
efficient (more easily satisfies the real time requirement), while in others 
the frequency domain approach is faster. In the latter situations, it is the 
FFT algorithm for computing the Fourier transforms that enables the 
superiority of frequency-domain implementations. Because complexity 
considerations only express how algorithm running-time increases with 
system parameter choices, we need to detail both implementations to 
determine which will be more suitable for any given filtering problem. 
Filtering with a difference equation is straightforward, and the number of 
computations that must be made for each output value is 2 (p + q). 
Exercise: 


Problem: 


Derive this value for the number of computations for the general 
difference equation. 


Solution: 


We have p + q+ 1 multiplications and p + gq — 1 additions. Thus, the 
total number of arithmetic operations equals 2 (p + q). 


